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[Type of the Document] Specification 
[Title of the Invention] 

USER INTERFACE APPARATUS, USER INTERFACE METHOD , GAME 
APPARATUS, AND PROGRAM STORAGE MEDIUM 

5 

[What Is Claimed Is:] 

[Claim 1] A user interface apparatus comprising: 
a first sensor attached to a first portion of a 
body of a user; 

10 a second sensor attached to a second portion 

different from the first portion; 

means for generating action information of the 
second portion on the basis of a relative position of 
the second portion with respect to the first portion, 
15 which is detected by said first and second sensors; and 
determination means for determining a user 
instruction corresponding to the generated action 
information _ 

[Claim 2] The apparatus according to claim 1, 
20 wherein the first portion is a head. 

[Claim 3] The apparatus according to claim l or 2, 
wherein the second portion is a hand. 

[Claim 4] The apparatus according to any one of 
claims 1 to 3, wherein said first sensor detects a 
25 location/posture of the first portion. 
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[Claim 51 The apparatus according to any one of 
claims 1 to 4, wherein said second sensor detects a 
location and posture of the second portion. 

[Claim 6] The apparatus according to any one of 
claims 1 to 5, wherein the action information includes 
information which pertains to a state change of the 
second portion with respect to a location of the first 
portion, and information which pertains to a location 
change velocity of the state change. 

rciaim 71 The apparatus according to any one of 
claims 1 to 6, wherein the action information includes 
information which pertains to a state change of the 
second portion with respect to a location or 
location/posture of the first portion, and information 
IS which pertains to a location change acceleration of the 
state change. 

[Claim 8] The apparatus according to any one of 
claims I to 7, wherein the action information includes 
information which pertains to a posture of the second 
portion with respect to a posture of the first portion. 

[Claim 9] The apparatus according to any one of 
claims 1 to 8. wherein the action information includes 
information which pertains to a moving direction of a 
location of the second portion with respect to a 
25 posture of the first portion. 

[Claim 10] The apparatus according to any one of 
claims 1 to 9, further comprising: 



20 
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means for storing a value of the relative 
position of the second portion with respect to the 
first portion, and a plurality of state values which 
are defined in advance as a result of transition of the 
5 value ; and 

means for storing a plurality of different user 
instruction values corresponding to the plurality of 
state values. 

[Claim 11] The apparatus according to any one of 
10 claims 1 to 10. wherein said determination means 
decomposes the determined user instruction into a 
plurality of instruction operands, and outputs the 
operands . 

[Claim 12] The apparatus according to any one of 
15 claims 1 to 11. wherein when said generation means 
determines that a relative relationship between a 
location /posture of a head detected by said first 
sensor, and a location/posture of a hand detected by 
said second sensor indicates an action of the user 
whose line of sight is pointing to a predetermined 
portion of the hand, 

said determination means outputs a user 
instruction for outputtlng an operation guidance. 

[Claim 13] The apparatus according to any one of 
claims 1 to 12, further comprising a third sensor for 
detecting a bent angle of a finger. 

[Claim 14] A game apparatus comprising: 
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a first sensor for detecting a location/posture 
of a head of a player; 

a second sensor for detecting a location/posture 
of a hand or arm; 

5 means for estimating an action of the player on 

the basis of a relative location/posture of the hand or 
arm with respect to the location/posture of the head, 
which are detected toy said first and second sensors; 
and 

10 means for outputting a player command 

corresponding to the estimated action. 

[Claim 15] The apparatus according to claim 14, 
further comprising display means for displaying an 
image of a game scene in front o£ the head of the 
15 player for the player. 

[Claim 16] The apparatus according to claim 14 or 
15. wherein said display means is a head-mounted 
display . 

[Claim 17] The apparatus according to any one of 
20 claims 14 to 16, wherein three steps including 

preparation, execution, and return steps, are prepared 
for the player command, and the steps are respectively 
defined as: 

a preparation action step for the player command 
25 when the location of the hand of the player moves to a 
position behind a position in front of a face of the 
player; 
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an execution step for the player command when the 
location of the hand of the player moves forward from a 
rear position after the preparation action; and 

a return action step for the player command when 
5 the location of the Hand of the player returns to a 
position of the face of the player after the forward 
movement . 

[Claim 18] A user interface method for outputting 
a user instruction to a predetermined apparatus or 

10 pro gram , comprising: 

the step of detecting a location of a first 
portion of a body of a user and a location of a second 
portion different from the first portion using first 
and second sensors attached to the user; and 

15 the step of determining a user instruction on the 

basis of a relative position of the second portion with 
respect to the first portion, which are detected by 
said first and second sensors, and outputting the 
determined user instruction to the apparatus or program. 

-° [Claim 19] The method according to claim 18 , 

wherein said first and second sensors respectively 
detect locations /postures of the first and second 
portions . 

[Claim 20] The method according to claim 18 or 19 , 
5 wherein the first portion is a head of the user, and 
the second portion is a hand of the user. 
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[Claim 21] The method according to any one of 
claims 18 to 20, further comprising the step of 
detecting information which pertains to a state change 
of the second portion with respect to a location or a 
5 location/posture of the first portion, and information 
which pertains to a location change velocity of the 
state change. 

[Claim 22] The method according to any one of 
claims 18 to 20, further comprising the step of 
10 detecting information which pertains to a state change 
of the second portion with respect to a location or a 
location/posture of the first portion, and information 
which pertains to a location change acceleration of the 
state change ♦ 

15 [Claim 23] The method according to any one of 

claims IS to 22, further comprising the step of 
detecting a posture of the second portion with respect 
to a posture of the first portion* 

[Claim 24] The method according to any one of 

20 claims 18 to 23 , further comprising the step of 
detecting a moving direction of a location of the 
second portion with respect to a posture of the first 
portion. 

[Claim 25] The method according to any one of 
25 claims 18 to 24, further comprising the step of storing 
a value of the relative position of the second portion 
with respect to the first portion, and a plurality of 
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user instructions which are defined in advance as a 
result of transition of the value. 

t Claim 26] The method according to any one of 
claims 18 to 25 , wherein the output step includes the 
5 step of decomposing the determined user instruction 
into a plurality of instruction operands, and 
outputting the operands. 

[Claim 27] The method according to any one of 

i 

claims 18 to 26, wherein when it is determined that a 
10 relative relationship between a location/posture of a 
head detected by said first sensor, and a 
location/posture of a hand detected by said second 
sensor indicates an action of the user whose line of 
sight is pointing to a predetermined portion of the 
15 hand, 

the output step includes the step of outputting a 
user instruction for outputting an operation guidance. 

[Claim 28] The method according to any one of 
claims 19 to 28, further comprising the step of 
20 detecting a bent angle of a finger from a third sensor. 

[Claim 29] A computer readable storage medium, 
which stores a program that makes a computer to perform 
a user interface method according to any one of claims 
18 to 28. 

25 [Claim 30] A game apparatus for displaying a CG 

image in front of a field of view of a playex. 
comprising: 
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a first sensor for detecting a location of a 
first portion of a body of the player; 

a second sensor for detecting a location of a 
second portion of the player, which is different from 
5 the first portion; and 

game progress means for proceeding with a game by 
determining a command the player wants to input on the 
basis of a relative position of the second portion with 
respect to the first portion, which are detected by 
10 said first and second sensors, and executing the * 
determined command _ 

[Claim 31] The apparatus according to claim 30, 
wherein said first and second sensors respectively 
detect locations /postures of the first and second 
15 portions, 

[Claim 32] The apparatus according to claim 30 or 
31, wherein the first portion is a head of the player, 
and the second portion is a hand of the player. 

[Claim 33] The apparatus according to any one of 

20 claims 30 to 32, further comprising means for detecting 
information which pertains to a state change o£ the 
second portion with respect to the location or a 
location/posture of the first portion, and information 
which pertains to a location change velocity of the 

25 state change, and 

wherein the command is generated in further 
consideration of the detected velocity . 
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[Claim 34] The apparatus according to any one of 
claims 30 to 33 , further comprising means for detecting 
information which pertains to a state change of the 
second portion with respect to the location or a 
5 location/posture of the first portion/ and information 
which pertains to a location change acceleration of the 
state change, and 

wherein the command is generated in further 
consideration of the deteated acceleration . 
10 [Claim 35] The apparatus according to any on© of 

claims 30 to 34, further comprising means for detecting 
a posture of the second portion with respect to a 
posture of the first portion, and 

wherein the command is generated in further 
15 consideration of the detected posture of the second 
portion . 

[Claim 36] The apparatus according to any one of 
claims 30 to 35, further comprising means for detecting 
a moving direction of the location of the second 

20 portion with respect to a posture of the first portion, 
and wherein the command is generated in further 
consideration of the detected moving direction of the 
location of the second portion. 

[Claim 37] The apparatus according to any one of 

25 claims 30 to 36, further comprising a memory for 

storing a value of the relative position of the second 
portion with respect to the first portion, and a 
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plurality of player commands which are defined in 
advance as a result of transition of the value. 

[Claim 38] The apparatus according to any one of 
claims 30 to 37, wherein said game progress means 
decomposes the determined player command into a 
plurality of operands, and executes the command in 
accordance with the plurality of decomposed operands. 

[Claim 39] The apparatus according to claim 33, 
wherein when a relative relationship between a 
location/posture of a head detected by said first 
sensor, and a location/posture of a hand detected by 
said second sensor indicates that a line of sight of 
the player is pointing to a predetermined portion of 
the hand, 

15 said game progress means generates a player 

command for outputting an operation guidance. 

[Claim 40] The apparatus according to any one of 
claims 30 to 39. further comprising a third sensor for 
detecting a bent angle of a finger. 
20 [Claim 41] The apparatus according to any one of 

claims 1 to 9. wherein the action information further 
includes geometric information of the first and second 
portions in a reference coordinate system. 

(Claim 42] The apparatus according to any one of 
25 claims 14 to 17, wherein said estimation means extracts 
geometric information of the first and second portions 
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in a reference coordinate system, and estimates a 
player action on the basis of the geometric information 

[Claim 43] The method according to any one of 
claims 18 to 28, wherein the detection step includes 
5 the step of extracting geometric information of the 
first and second portions in a reference coordinate 
system. 

[Claim 44] The apparatus according to any one of 
claims 30 to 40, wherein said game progress means 

10 further extracts geometric information of the first and 
second portions in a reference coordinate system, 
estimates an action of the player on the basis of the 
geometric information, and determines the command the 
player wants to input on the basis of the estimated 

1 5 action . 

[Detailed Description of the Invention] 
[0001] 

[Technical Field to Which the Invention Belongs] 

The present invention relates to a user interface 
20 apparatus and method, and a game apparatus, which 
generate an instruction (or command) by adequately 
detecting player's (user's) actions in, e.g.. an action 
game. 

[0002] 
25 [Prior Art) 

A current personal computer system uses as a user 
interface a pointing device such as a mouse, track pad. 
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or the like. However, the user must hold the mouse and 
slide it on a given surface. On the other hand, the 
user must rub against the surface of the track pad with 
his or her hand. Thus, these pointing devices limit 
5 user's actions. A GUI (Graphical User Interface) used 
in the personal computer system or the like is that for 
a two-dimensional space, and is not suitable for that 
in a three-dimensional space. 
[0003] 

10 For tfa ese reasons, it is a common practice in the 

technical field of VR (Virtual Reality) or AR 
(Augmented Reality) to input commands to the system by 
switch operations of an input device which is held by a 
user's (player's) hand and has switch buttons, and the 

15 like. 
[0004] 

[Problems That the Invention Is to Solve] 

In the prior art in which commands are input by 
switch operations on the input device having button 

2 0 switches and the like, the number of types or number of 
commands (instructions) is limited by the number of 
buttons. If the number of types or number of commands 
(or instructions) is increased, the number of buttons 
increases inevitably. As a result, the input device 

25 becomes large in size, and the load on the user 

(player) becomes heavier as he or she must learn the 
button positions. 
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[0005] 

Learning the button positions imposes a heavy 
load on the user since the command contents do not 
match the button positions. To put it differently, 
expressing various command contents (e.g., "forward 
movement", "backward movement", "stop", and the like) 
by one operation, i.e., depression of a button (or a 
switch) is difficult if not impossible. 
[0006] 

On the other hand, in the VR (Virtual Reality) or 
AR (Augmented Reality) field, a device for simulating 
user's (player's) hand actions has been proposed. For 
example, in one technique, a sensor for detecting the 
bent angle of a joint of a finger is attached to a hand 
15 of the user, and a CG (computer graphic) image is 

generated in correspondence with the bent angle of the 
finger detected by that sensor. However, this 
technique aims at simulating hand actions of the user, 
and it is impossible to apply this technique to that 
for recognizing the user (player) instructions (or 
commands) in practice. 

[0007] 

In this technique, for example, when the user 
stretches the arm forward, a CG image with the arm 
25 stretched forward is generated, and display of such 
image can be interpreted to be a result of a user 
instruction or forward stretching of the arm in the 



20 



_ it 
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broad sense. However, If user commands are generated 
by only hand actions, every hand actions are unwantedly 
interpreted as commands, and such interface has poor 
reliability, 
5 [0008] 

The present invention has been made to solve the 
conventional problems and has as its object to provide 
a user interface apparatus, user interface method, and 
game apparatus, to which the user (player) can easily 
10 sensuously become accustomed, and which can accurately 
recognize instructions (commands) that the user 
(player) intended. 
[0009] 

[Means of Solving the Problems] 
15 In order to achieve the above object, a user 

interface apparatus according to claim 1 comprises: 
[0010] 

a first sensor attached to a first portion of a 
body of a user; 

20 a second sensor attached to a second portion 

different from the first portion; 
[0011] 

means for generating action information of the 
second portion on the basis of a relative position of 
25 the second portion with respect to the first portion, 
which is detected by the first and second sensors; and 
[0012] 
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determination means for determining a user 
instruction corresponding to the generated action 
information. 
[0013] 

5 Also r a game apparatus according to claim 14 

comprises : 
[0014] 

a first sensor for detecting a location/posture 
of a head of a player; 
10 [0015] 

a second sensor for detecting a location/posture 
of a hand or arm; 
[0016] 

means for estimating an action of the player on 
15 the basis of a relative location/posture of the hand or 
arm with respect to the location/posture of the head, 
which are detected by the first and second sensors? and 
[0017] 

means for output ting a player command 
20 corresponding to the estimated action. 
[0018] 

Likewise, a user interface method for outputting 
a user instruction to a predetermined apparatus or 
program according to claim 18 comprises: 
25 [0019] 

the step of detecting a location of a first 
portion of a body e>f a user and a location of a second 
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portion different from the first portion using first 
and second sensors attached to the user; and 

[0020] 

the step of determining a user instruction on the 
5 basis of a relative position of the second portion with 
respect to the first portion, which are detected by the 
first and second sensors, and outputting the determined 
user instruction to the apparatus or program. 
10021] 

10 In claims 1, 14 and 18, when the user (player) 

expresses his or her intention or instruction by his or 
her own action, he or she often expresses it by a 
combination of actions of a plurality of his or her 
body portions. Such combination of actions can be 

15 estimated by detecting the location of the second 
portion relative to the first portion, as described 
above , 
[0022] 

The user or player normally expresses an action 
20 by a relative location with respect to his or her 

field-of-view direction. The f ield-o£-view direction 
is roughly determined by the head location. Hence, 
according to claim 2, which is a preferred aspect of 
the present invention, the first portion is a head* 
25 [0023] 

According to claim 3, which is a preferred aspect 
of the present invention, the second portion is a hand. 
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This is because it is easiest for the user to express 
&n action by hand. 
[0024] 

According to any one of claim 4, 5, 19 and 20, 
which is a preferred aspect of the present invention, 
the first (second) sensor detects a location or 
location/posture of the first (second portion. 
[0025] 

User's (player's) action is preferably recognized 
as a state change. Hence, according to claim 6 or 21, 
which is a preferred aspect of the present invention, 
information which pertains to a state change of the 
second portion with respect to a location of the first 
portion, and information which pertains to a location 
15 change velocity of the state change are detected. 
[0026J 

According to claim 7 or 22, which is a preferred 
aspect of the present invention, information which 
pertains to a state change of the second portion with 
20 respect to a location of the first portion, and 
information which pertains to a location change 
acceleration of the state change are detected. 
[0027] 

According to claim 8 or 23, which is a preferred 
25 aspect of the present invention, information which 
pertains to a posture of the second portion with 
respect to a posture of the first portion is detected. 
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[0028] 

According to claim 9 or 24, which is a preferred 
aspect of the present invention, information which 
pertains to a moving direction of a location of the 
second portion with respect to a posture of the first 
portion is detected. 
[0029] 

Various methods of analyzing an action state are 
available. According to claira 10 or 25 as a preferred 
aspect of the present invention, a value of the 
relative position of the second portion with respect to 
the first portion, a plurality of state values which 
are defined in advance as a Result of transition of the 
value, and a plurality of different user instruction 
values corresponding to the plurality of state values 
are stored. 
[0030] 



A user instruction or command is composed of a 
plurality of portions in practice, and it is 
advantageous for a program to individually execute 
those portions of the instruction or command. Hence, 



which is a preferred 



according to claim 12 or 26, 
aspect of the present invention, the determined user 
instruction or command is decomposed into a plurality 
of instruction operands, and 
[0031] 



the operands are output. 
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15 



between a location /posture 



outputting an operation 



According to claim 12 or 27, which is a preferred 
aspect of the present invention, when it is determined 
that a relative relationship 

of a head detected by the first sensor, and a 
location/posture of a hand detected by the second 
sensor indicates an action o:: the user whose line of 
sight is pointing to a predetermined portion of the 
hand, a user instruction for 

guidance (so-called on-line ^IELP) is generated. 
[0032J 

According to claim 13 or 28 r which is a preferred 
aspect of the present invention, a third sensor for 
detecting a bent angle of a finger is further used. 
£0033] 

When the present invention is applied to the game 
apparatus, a display device is preferably used. Hence, 
the game apparatus according 



game scene in front of the 



to claim 15 as a preferred 
aspect of the present invention comprises display means 
for displaying an image of a 
20 head of the player for the player. Furthermore, 

according to claim 17 , the display means is preferably 
a head-mounted display. 
[0034] 

When the present invention is applied to the game 
25 apparatus, three steps including preparation, execution, 
and return steps, are preferebly prepared for the 
player command. According to claim 17, which is a 
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preferred aspect of the present invention, the steps 
are respectively defined as: 
[0035J 

a preparation action step for the first player 
command when the location of the hand of the player 
moves to a position behind a position in front of a 
face of the player; 
[0036] 

an execution step for the first player command 
when the location of the hand moves forward from a rear 
position after the preparation action; and 
[0037] 

a return action step for the first player command 
when the location of the hand returns to a position of 
15 the face of the player after the forward movement. 
[0038] 

Note that the above object can also be achieved 
by a program storage medium that stores a computer 
program for implementing the aforementioned user 
interface method, as defined in claim 29. 
[0039] 

[Embodiments ] 

A game apparatus to which a user interface of the 
present invention is applied will be described in 
detail hereinafter with reference to the accompanying 
drawings. As will be apparent from the following 
description of the embodiment, the user interface of 



20 



25 
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the present invention can be applied to a personal 
computer system or workstation system for providing a 
VR or AR environment to the user in addition to the 
game apparatus by modifying the present invention 
5 within the scope of the invention. 
[0040] 

Fig- 1 shows a user 1000 who manipulates the game 
apparatus of this embodiment. A magnetic sensor 100 as 
an example of a location/posture detection means is 1 

10 attached to the head of the user 1000, and a magnetic 
sensor 200 is also attached to a hand. The magnetic 
sensors output electrical signals including information 
of a three-dimensional location (x, y, z) and posture 
(roll, pitch, yaw) of! the portions where they are 

15 attached. 
[0041] 

Fig, 2 shows the game apparatus system o£ this 
embodiment. Referring to Fig. 2, the game system of 
this embodiment has a command generation unit 2000 for 

20 receiving signals from the two sensors and converting 
them into a command, and a game apparatus 3000* The 
command generation unit 2000 analyzes an action of the 
user 1000 on the basis of the output signals from the 
sensors 100 and 200, and generates a command in 

25 accordance with the analysis^result . The generated 
command is sent to the game apparatus 3000 , which 
executes the command, i.e., the game progresses. 
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[0042] 

The arrangement and operation of the command 
generation unit 2000 will be described below. As shown 
in Fig- 2, the unit 2000 has location/posture 
5 measurement sections 2001a and 2001b, action analysis 
section 2002. and command conversion section 2003. 
[0043] 

The location/posture measurement sections 2001a 
and 2001b convert electrical signals input from the 

10 location sensors 100 and 200 into six each coordinate 
values representing the locations /postures of the 
individual portions, and send them to the action 
analysis section 2002, The location/posture 
measurement sections 2001a and 2001b have internal 

15 clocks to measure a measurement time t in addition to 
the six each coordinate values of the 
locations/postures , and output the measured times to 
the command generation unit 2000. 
[0044] 

20 The action analysis section 2002 analyzes an 

action in consideration of not only the absolute 
locations /postures of the head and hand but also their 
relative location/posture. , This is for the following 
reason. That is, a person normally expresses his or 

25 her own intention by actions of a plurality of body 

portions. In other words, when the location/posture of 
only one portion are detected and the user's intention 
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10 



Is estimated based on them, many errors are produced 
even when the detected location/posture have high 
precision. Fox example, as for a forward stretching 
action of the arm. a forward stretching action of the 
arm with the head facing forward has a meaning 
different from that with the head facing another 
direction. For example, if the action "to stretch the 
arm forward" means "attack", since attack is normally 
done with the player's head facing the target (i.e., in 
the stretching direction of the arm), the player may 
intend an action other than attack if he or she 
stretched the arm with the head facing another 
direction. Hence, if no posture is taken into 
consideration, a forward stretching action of the arm 
15 with the head facing another direction is highly likely 
to be erroneously recognized as attack. For this 
reason, in this embodiment, the estimation precision of 
the user's intention is improved by taking relative 
location/posture (e.g., the location/posture of the 
20 hand relative to those of the head) into consideration. 
[0045] 

Fig. 3 shows the relationship between the 
locations/postures of the head and hand In a reference 
coordinate system, and Fig. 4 shows the relationship 
25 between the locations /postures o£ the head and hand in 
a head coordinate system. Figs. 5 and 6 respectively 
show the head and hand coordinate systems. 
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[0046] 

Fig. 7 shows the functions of the action analysis 
section 2002. The functions shown in Fig. 7 are 
implemented by the control sequence according to the 
5 flow chart shown in Fig. 8. According to the flow 
chart shown in Fig. a, three-dimensional locations 
(Lhcad and L hand ) and postures PiV^t. and P han(1 ) of the 
head and hand are input from the location/posture 
measurement sections 2001 in step S2. and a location 
10 and posture {L'^t and P'n«» d ) of the hand relative to 
the head are computed based on the input locations and 
postures in step S4. More specifically, a coordinate 
transform matrix M^na from the reference coordinate 
system into the hand coordinate system is computed on 
15 the basis of the hand location and posture (L haad and 
Phand), a coordinate transform matrix M,^ from the 
reference coordinate system into the head coordinate 
system is computed on the basis of the location and 
posture (L heod and P h<Mld ) of the head, and based on these 
20 matrices, a coordinate transform matrix M'^ana from the 
head coordinate system into the hand coordinate system 
is computed with the following [Equation 1] : 
[0047] 

[Equation 1] 

^5 M'hand = Mjiana'Mheaa"* 

[0048) 
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where M h( ^ a ~ l is the Inverse matrix of M hBad . The 
relative location and posture (L f hand , P f hand ) of the 
hand with reference to the head (i.e., in the head 
coordinate system) can be easily derived from M'^^ 
5 in step S6, change velocity V han d (time derivative 

of L'hani) of the hand location L'hond, and acceleration 
Ahand (time derivative of velocity V h and) are computed. 
Note that the currently measured location and posture 
(L'handi P'nand) of the hand relative to the head are 
10 stored in a predetermined memory for the next velocity 
and acceleration computations in step S6. in this case, 
data for two previous measurements suffice- in this 
specification, a combination of information of the 
location, posture, time, and the like for the head, and 
15 those for the hand input from the location/posture 
measurement sections 2001 is called "primary 
information" # and the location and posture, and 
velocity and acceleration of the hand with reference to 
the head are called "secondary information". 
20 [0049] 

It is checked in step SB if "secondary 
information" satisfies a predetermined state transition 
condition so as to determine transition of state q>. 
The operation of the action analysis section 2002 upon 
25 applying the functions shown in Figs. 7 and 8 to an 
action (shooting) game will be explained below* In 
this shooting game, the following five states are 



PAGE 29(62 ' RCVD AT 10/2112005 1 1:34:33 AM [Eastern Daylight Time] ' SVR:USPTO£FXRF-6/30 ' DNIS:2734451 " CSID:202 530 1055 ' DURATION (mm-ss):09-08 



10/21/2005 11:37 FAX 202 530 1053 



FCH&S D.C. 



@030 

11-165709 



defined, and values 0 to 4 are assigned in advance to 
these states. 

0 = initial state 

1 = loading (load a gun) state 
5 2 => shooting state 

3 - defense (protect a player) state 

4 = operation guidance state 

[0050] 

In this game apparatus, an action is expressed by a 
10 change in state value. As shown in Fig. 9, actions 
defined in this game apparatus are as follows; 

Transition Action 

* * No action (no change in state value) 

15 0 ~* 1 Loading action 

1 0 Loading cancel action 

1 ~* 2 Firing action 

0 3 Defense action 

3 ~* 0 Defense cancel action 

20 0 4 Operation guidance action 

4 -* 0 Operation guidance cancel action 

Note that the initial state is restored a predetermined 
period of time elapsed after the firing state. The 
output from the action analysis section 2002 includes 
25 action state qp and three-dimensional information of the 
head and hand. 
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Fig. 10 is a flow chart showing the overall state 
transition condition determination process of the 
action analysis section 2002. More specifically, the 
action analysis section 2002 executes one of steps S100, 
5 S200, S300, S400. and S500 (respectively shown in 

detail in Figs. 11, 12. 13, 14. and 15) in accordance 
with the current state value cp. Each of steps S100, 
S200, S300, S40O, and S500 outputs the action state 
value 9 after transition, and three-dimensional 
10 information of the head and hand. 
[0051] 

The control sequence when the current state value 
= 0, i.e., "initial state" is currently set, will be 
explained below with reference to Fig, 11. There are 
15 four states to which the initial state can transit, 

i.e., state 0 (no transition), state 1 (loading state), 
state 3 (defense state), and state 4 (operation 
guidance state), in the game apparatus of this 
embodiment. In the control sequence shown in Fig. 11, 
20 to which of the four states the initial state is to 

transit is determined in consideration of the amount of 
backward movement of the hand with respect to the head 
in the back (z) direction, the deviation of the angle 
the back -and- forth direction (z-direction in Fig. 6) of 
25 the hand makes with the back direction (z-direction in 
Fig. 5) of the player's head, the deviation of the 
angle the back of the hand (y-direction in Fig. 6) 
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maKes with the back direction of the head, and the like. 
Hence, the following variables z hand , a%r and ct 2 must be 
considered in the control sequence shown in Pig. U« 
[0052] 

5 z,^: z- coordinate value of hand location with 

reference to head coordinate system (see Fig. 5) 
[0053J 

a x : angle z-axis of head coordinate system makes 
with that of hand coordinate system (see Fig. 6) 
10 [0054] 

a a : angle z-axis of head coordinate system makes 
with y-axis of hand coordinate system 
[0055] 

Assume that constant values C 0 i, C 02 , C 03 . and C 04 
15 are respectively: 

Coi = 0.0 (length) 

E0056] 

C 02 = 45° 

[0057] 
20 C 03 = 150° 

[0058] 

C 04 = 30° 

[0059] 

Then, if YES in steps S102 and S104, i.e., in 
25 case that 

Zhano > Coi and Cti < C 0 2. 
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in other words, if the player has moved the hand 
backward to a location behind the center of the head 
(C 0 i = 0.0) in the back direction of the head while the 
deviation of the angle the direction of the hand makes 
5 with the back direction of the head ig kept suppressed 
to be equal to or smaller than 45* , it la determined 
that the player wants to make a loading action. Then, 
the current time t is saved in a register t M in step 
S106, the current hand location Lhana is saved in a 
10 register L 0 i in step S108. and an action state value = 
1 (loading state) is output in step SI 10. 
[0060] 

On the other hand, if NO in steps S102 and S104, 
and YES in step S112, i.e., in case that 
15 [0061] 

Zhand 2= c ox and ai & c 02 and a 2 > C 03 , 
in other words, if the player has moved the hand to a 
location in front of the central position of the head 
while the deviation of the angle the direction of the 

20 hand makes with the back direction of the head is 45° 
or more, and an angle the direction perpendicular to 
the back of the hand makes with the back direction of 
the head is 150° or more, it is determined that the 
player wants to make a defense action. Then, the 

25 current time t is saved in a register t 03 in step S114, 
and an action state value = 3 (defense state) is output 
in step S116. 
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[0062] 

On the other hand, if NO in steps S102, S104, and 
S112, and YES in step S118, i.e. , in case that 
[0063] 

5 z *iand ^ C 01 and ai & Cqz and a 2 < C04, 

in other words, if the player has moved the hand to a 
location in front of the central position of the head 
while the deviation of the angle the direction of the 
hand makes with the back direction of the head is 45° 

10 or more, and an angle the direction perpendicular to 
the back of the hand makes with the back direction of 
the head is 30° or less, it is determined that the 
player wants an operation guidance (i.e., HELP). Then, 
the current time t is saved in a register to* in step 

15 S120, and an action state value = 4 (operation guidance 
state) is output in step S122. 
[0064] 

On the other hand, if NO in all steps S102, S104 r 
S112, and Slig, i.e., in case that 
20 [0065] 

Zhand * C 0 i and <*i a c 02 and C 0 4 s ct 2 *s C 0 3, 
it is determined that the player's action is not large 
enough to issue a command, and the initial state is 
maintained in step S124. 
25 [0066] 

The control sequence when the current state is 
the loading state (:= 1) will be explained below with 
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reference to Fig. 12. There are three states to which 
the loading state can transit, i.e., state 0 (initial 
state), state 1 (no transition), and state 2 (firing 
state), as shown in Fig. 9. In the control sequence 
5 shown in Fig. 12, a hand moving distance., d, hand angle 
a 3# and hand moving velocity v are referred to in 
addition to the aforementioned angle a x * Note that 
[0067] 

d: distance from hand location L 0 to current hand 
10 location L hand when state transits from 0 to 1 
[0068] 

ct 3 : angle a velocity vector V of hand malces with 
2 -axis of head coordinate system 
[0069] 

15 v: moving velocity of hand 

[0070] 

Assume that constant values Cu, C 12 , Cu, and a** 
are respectively: 

Cn = 60* (to be compared with a*) 

20 [0071] 

C 12 - 300 mm (to be compared with d) 

[0072] 

C13 = 120° (to be compared with ct 3 ) 

[0073] 

25 C i4 = 300 mm/ s (to be compared with v) 

[0074] 

Then , if 
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*l > C u 

±n step S201, in other words, the deviation o£ the 
angle the hand direction makes with the back direction 
of the head has exceeded 60°, it is determined that the 
5 player has made a loading action but does not want to 
fire. Then, the current tijne t is saved in a register 
tio in step S202, and an action state value = 0 is 
output in step S204. 
[0075] 

10 ° n the other hand, if NO in step S201 and YES in 

steps S206, S208, and S210, i.e., in case that 
[0076] 

<*i * Cu, d > Ci 2 , a 3 > Ci 3 , v > C14, 
in other words, if a change in direction of the hand 

15 with respect to the head is not large, the hand moving 
distance is equal to or larger than Ci 2 (e.g. , 300 mm), 
the angle the hand moving direction makes with the back 
direction of the head is equal to or larger than the 
angle C i3 (e.g., 120°) , and the hand moving velocity is 

20 equal to or higher than C 14 (e.g., 300 mm/s), it is 
determined that the player wants to fire. Then, the 
current time t is saved in a register t 12 in step S212, 
and an action state value = 2 (firing state) is output 
in step S214. 
25 [0077] 

On the other hand, if the above conditions are 
not satisfied, i.e., in case that 
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[0078] 

a x £ Cn and d £ C12 or a 2 =s C l3 , v £ C 14 , 
the loading state is maintained . 
[0079] 

5 When the current state is the firing state (= 2). 

the oontrol sequence shown in Pig. 13 is executed. In 
the example shown in Fig. 9, the firing state can 
transit to either the initial state or firing state (no 
transition). According to the control sequence shown 

10 in Fig* 13 , if time At elapsed after the loading state 
transits to the firing state is smaller than a constant 
value T 3 (e.g., 200 msj (NO in step S302), the firing 
state is maintained in step S308. On the other hand, 
if time At is equal to or longer than T 3 , the current 

15 time is saved in step $304, and the state is returned 
to the initial state to output that state value in step 
S306- 
[0080] 

in the example in Fig. 9, the initial state is 
20 unconditionally restored the predetermined period of 

time (T 2 ) after the beginning of shooting. Such simple 
state transition to the initial state is designed for 
the sake of simplicity upon explaining the rules of the 
game of the game apparatus of this embodiment. If the 
25 game has a complex flow (scenario), more states should 
be defined in correspondence with that complexity.' In 
such case, more states in addition to the initial state 
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may be defined as those to which the firing state can 

transit. 

[0081] 

When the current state is the defense state (= 3), 
5 the control sequence shown in Fig. 14 is executed- In 
the example shown in Pig. 9, states to which the 
defense state can transit are only the initial state 
and the defense state (no transition) . According to 
the control sequence shown in Fig. 14, it is checked in 
10 step S402 if an angle the posture of the hand makes 

with that of the head is smaller than a predetermined. 

value C 3 i (e.g., 150°). That is, if 

[0082] 

15 in other words , if an angle the direction of the back 
of the hand makes with the back direction of the head 
is set to be less than 150°, it is determined that the 
defense state is canceled, and the current state is 
returned to the initial state and the current time is 

20 saved in step S404. Furthermore, the initial state (= 
0) is output in step S406. 
[0083] 

On the other hand, if NO in step S402, but if it 
is determined in step S407 that time At elapsed after 
25 the previous state transited to the defense state is 
equal to or longer than a predetermined time T 3 , the 
flow advances to step S404 to return the current state 
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to the initial state like in the case wherein a 
sufficiently long time has elapsed from the firing 
state. 
[0084] 

5 Furthermore, if NO in steps S402 and S407, i.e., 

in case that 
[00S5] 

ct 2 & C 3 i and AT s T ]y 
the defense state is maintained. 
10 [0086] 

If the state transits to "operation guidance 
state" in step S122 in Fig. n, i.e., in case that 
[0087] 

z*ond * Coi and cti > c g3 and a 2 ^ C 0 «, 
15 in other words , if the player moves the hand to a 

location in front of the central position of the head 
in the back direction of the head while the deviation 
of the angle the direction of the hand makes with the 
back direction of the head is 45° or more, and an angle 
20 the direction perpendicular to the back of the hand 
makes with the back direction of the head is 30° or 
less, this means that the player is watching the back 
or joint of the hand* In such case, since a state 
value = 4 is output to the game apparatus 3000, the 
25 apparatus 3000 displays a HELP window on the 

predetermined display device. Note that this display 
device is preferably a head-mounted display device 
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(HMD) . If the HMD is used, the player need not move 
the head upon watching the HELP window, and the control 
sequence (Fig. 15) for maintaining the "operation 
guidance state" need not be complicated. 
5 [0086J 

Note that the HELP window may be either two- or 
three-dimensional display, or may be displayed in 
either a VR or AR environment, as long as it matches 
the current environment. In this embodiment, sinoe the 

10 head location /posture sensor is attached to the player, 
it is preferable to display the HELP window in an AR 
environment in terms of efficient use of the sensor 100. 

Various modifications of the present invention 
can be made. 

15 [0089] 

Modification 1: 

[0090] 

In the above embodiment, commands and states 
shown in Fig, 9 are used. However, the present 

20 invention is not limited to such specific commands and 
states. That is, the user interface of the present 
invention is not limited to a game apparatus 
environment* The present invention can be applied to 
any other environments as long as the user's or 

25 player's actions axe required to be used in place of 
commands (or instructions) upon presenting CG data to 
the user or player. 
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[0091] 

For example, the present invention may be applied 
to a sign language recognition user interface, a user 
interface for manipulating home electronic products, a 
5 user interface for manipulating industrial machines 

(special vehicles for construction work, factory lines), 
a user^ interface for physically handicapped person 
assist apparatuses (assist bed, motored wheelchair), 
and the like. 
10 [0092] 

Modification 2» 

[0093] 

When the present invention is applied to a game 
apparatus, the game to which the present invention is 
15 applied is not limited to the rules shown in Fig. 9. 
The present invention can be applied to more or less 
states (commands). Also, state transition is not 
limited to Fig. 9- 
[0094] 

20 Modification 3: 

[0095] 

in the game system of the above embodiment, since 
the action states have one-to-one correspondence with 
commands or instructions , the command conversion 
25 section 2003 can directly convert an action state into 
a command. However, the present invention is not 
limited to such specific system. That is, the command 

i 
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conversion unit 2003 is designed to improve 
compatibility to the game apparatus 3000 connected to 
the unit 2000, In other words, the game apparatus 3000 
can be an existing versatile game apparatus . The 
5 existing game apparatus has an existing command system. 
The command conversion section 2003 is designed to have 
a table for receiving outputs (state, velocity , 
acceleration) from the action analysis section 2002, 
and converting them into a command system for the game 

10 apparatus 3000. This table is rewritable, and the 

table contents are rewritten in correspondence with the 
game apparatus 3000 used. In this way, the unit 2000 
can be applied to a plurality of different types of 
game apparatuses having different command systems 

15 without changing the arrangement and operation of the 
unit 2000. 
[0096] 

Modification 4: 

[0097] 

20 In the above embodiment, sensors are attached to 

the head and hand. However, the present invention is 
not limited to this. That is, to achieve the objective 
for issuing a command by an action, the sensor may be 
attached to the fingertip, arm, leg, thigh, knee, or 

25 shin in place of the hand. For example, as an example 
of a sensor to be attached to finger joints, a so- 
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called "glove type hand joint angle measurement device- 
has been put into practice. 
[0096] 

Modification 5: 

5 [0099] 

In the above embodiment, the command conversion 
section 2003 receives an action identifier from the 
analysis section 2002, and outputs a commana signal 
corresponding to an action- Upon receiving this 

10 command, the command signal system of the programs and 
apparatuses (game apparatus and the like) that use the 
command signal is defined in the game apparatus 3000. 
However, the present invention is not limited to such 
specific command signal. As the command signal, the 

15 following formats are available. That is, 
[0100] 

•to output a command signal as an electrical or 
optical signal; 
[0101] 

20 'to implement command conversion as a function 

program on a computer, and output data to be written in 
a predetermined storage area by the function program to 
the conversion unit 2003 as a command signal; 
[0102] 

* 5 "to implement command conversion as a function 

program on a computer, and launch a callback function 
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which corresponds to a command in advance to use it as 

a command signal output ; and 

[0103] 

•to implement command conversion as a function 
5 program on a computer , and generate an interrupt signal 
which corresponds to a command in advance to use it as 
a command signal output* 
[0104] 

Modification 6: 

10 [0105] 

In the above embodiment , magnetic sensors are 
used as the sensors. Alternatively, the 
location/posture may be measured using an ultrasonic 
wave, mechanically, or by means of an image process. 

15 [0106] 

[Effects of the Invention] 

As described, according to the present invention, 
a user interface apparatus, user interface method, and 
game apparatus, to which the user (player) can easily 

20 sensuously become accustomed, and which can accurately 

# 

recognize instructions (commands) that the user 

4 

(player) intended can be provided. 
[Brief Description of the Drawings] 
[Fig. 1] 

2 5 A view for explaining the layout of sensors 

attached to a player who joins in a game system 
according to an embodiment of the present invention. 
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j 

[Fig. 2] \ 

I 

A block diagram for explaining the arrangement of 
the game system according to the embodiment of the 
present invention . 
5 [Fig. 3] 

i 
i 

A view for explaining the relationship between 
the head and hand coordinate systems when viewed from a 
reference coordinate system as the center. 
[Fig. 4] 

10 * view for explaining the relationship between 

j 

the head and hand coordinate system when viewed from 
the head coordinate system as the center _ 
[Fig. 5] 

A view for explaining setups of a coordinate 
IS system of a head sensor '100 according to the scheme 
shown in Fig, 4. 

[Fig. 6] : 

i 

A view for explaining setups of a coordinate 
system of a hand sensor 200 according to the scheme 
20 shown in Fig. 4. 
[Fig, 7] 

i 

A block diagram fot functionally explaining the 
arrangement of a command generation unit 2000. 
[Fig. 8] ; 

25 A fl ow chart for explaining the overall sequence 

of an action analysis section 2002. 
[Fig. 9] 
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A view for explaining state transition in the 
game system according to the embodiment of the present 
invention. 
[Fig. 10] 

5 A flow chart for explaining details of step SB in 

Flg # 6. 

[Fig. 11] 

A flow chart for explaining details of step SI 00 
in Fig. 10. 
10 [Fig. 12] 

A flow chart for explaining details of step S200 
in Fig. 10* 
[Fig. 13] 

A flow chart for explaining details of step S300 
15 in Fig. 10. 
[Fig. 14] 

A flow chart for explaining details of step S400 
in Fig. 10. 
[Fig, 15] 

20 A flow chart for explaining details of step S500 

in Fig. 10. 
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I Type of the Document] Abstract 
[Abstract] 

t Problem] To provide a user interface apparatus to 
which the user (player) can easily sensuously become 
5 accustomed, and which can accurately recognize 

instructions (commands) that the user (player) intended. 
[Solving Means] The user interface apparatus has a 
location/posture sensor (100) attached to the head of 
the user, and a location/posture sensor (200) attached 

10 to his or her hand, analyzes hand action on the basis 
of the relative position of the location/posture of the 
hand with respect to those of the head, and estimates 
the current action state from the analysis result. The 
apparatus then determines a user instruction (player 

15 command) on the basis of the action state. 
[Selected Drawings] Fig. 7 



i 
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